Skip to content

Improve stderr output while running benchmarks #299

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
fitzgen merged 1 commit into from
Dec 19, 2025
Merged

Improve stderr output while running benchmarks #299

fitzgen merged 1 commit into from
Dec 19, 2025

Conversation

@fitzgen
Copy link
Member

@fitzgen fitzgen commented Dec 18, 2025

I originally pushed back against making the output too complicated, and I still believe we shouldn't pull in any fancy terminal status bar libraries or use terminal escape sequences and all that, but I do think that getting an idea of how longer the benchmarks would take is really good UX.

@posborne mind taking a look at this?

@fitzgen
Improve stderr output while running benchmarks
cd79573 
I originally pushed back against making the output too complicated, and I still
believe we shouldn't pull in any fancy terminal status bar libraries or use
terminal escape sequences and all that, but I do think that getting an idea of
how much longer the benchmarks will take is really good UX.
@fitzgen fitzgen force-pushed the improve-stderr-output branch from dab2db3 to cd79573 Compare December 18, 2025 22:24
@fitzgen
Copy link
Member Author

fitzgen commented Dec 18, 2025

Example output:

$ cargo run --release -- benchmark -m insts-retired --benchmark-phase compilation --processes 40 --iterations-per-process 1 -e /tmp/value-set.so -e /tmp/main.so -- benchmarks/shootout/shootout-keccak.wasm 
    Finished `release` profile [optimized] target(s) in 0.12s
     Running `target/release/sightglass-cli benchmark -m insts-retired --benchmark-phase compilation --processes 40 --iterations-per-process 1 -e /tmp/value-set.so -e /tmp/main.so -- benchmarks/shootout/shootout-keccak.wasm`

Running 80 total iterations (2 engines * 1 benchmarks * 40 processes * 1 iterations per process)

[Done] [Elapsed    ] [Est. Rem.  ]
[  1%] [00h:00m:00s] [00h:00m:00s] ..........
[ 14%] [00h:00m:10s] [00h:01m:03s] ..........
[ 26%] [00h:00m:26s] [00h:01m:13s] ..........
[ 39%] [00h:00m:36s] [00h:00m:57s] ..........
[ 51%] [00h:00m:51s] [00h:00m:49s] ..........
[ 64%] [00h:01m:01s] [00h:00m:35s] ..........
[ 76%] [00h:01m:11s] [00h:00m:22s] ..........
[ 89%] [00h:01m:23s] [00h:00m:11s] ..........

Finished benchmarking in 00h:01m:40s

compilation :: instructions-retired :: benchmarks/shootout/shootout-keccak.wasm

  Δ = 5143.55 ± 550.12 (confidence = 99%)

  value-set.so is 1.01x to 1.01x faster than main.so!

  [684043 685322.97 687123] main.so
  [679290 680179.43 685349] value-set.so

posborne
posborne approved these changes Dec 19, 2025
View reviewed changes
Copy link
Collaborator

@posborne posborne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I like this. Definitely an improvement over the dots which are not particularly informative at present. I agree that avoiding fancier options is probably less likely to introduce problems in the variety of places the benchmarks might be run.

@fitzgen fitzgen merged commit 8b83e79 into bytecodealliance:main Dec 19, 2025
17 checks passed
@fitzgen fitzgen deleted the improve-stderr-output branch December 19, 2025 17:08
@fitzgen
Copy link
Member Author

fitzgen commented Dec 19, 2025

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@posborne posborne posborne approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fitzgen @posborne